Diffuser: Efficient Transformers with Multi-Hop Attention Diffusion for Long Sequences

Efficient Transformers have been developed for long sequence modeling, due to their subquadratic memory and time complexity. Sparse Transformer is a popular approach improving the efficiency of by restricting self-attention locations specified predefined sparse patterns. However, leveraging sparsity may sacrifice expressiveness compared full-attention, when important token correlations are multiple hops away. To combine advantages both transformer full-attention Transformer, we propose Diffuser, new state-of-the-art efficient Transformer. Diffuser incorporates all interactions within one attention layer while maintaining low computation costs. The key idea expand receptive field using Attention Diffusion, which computes multi-hop based on paths between corresponding disconnected tokens, besides among neighboring tokens. Theoretically, show as universal approximator sequence-to-sequence investigate its ability approximate analyzing graph expander property from spectral perspective. Experimentally, effectiveness with extensive evaluations, including language image Long Range Arena (LRA). Evaluation results that achieves improvements an average 0.94% text classification tasks 2.30% LRA, 1.67x savings benchmarks, demonstrates superior performance in aspects.